[ZEPPELIN-1500] pyspark completion doesn't work. by astroshim · Pull Request #1465 · apache/zeppelin

astroshim · 2016-09-27T16:08:36Z

What is this PR for?

@fanlu reported bug on #530.
This PR fixes bug of checking condition in PySparkInterpreter.

What type of PR is it?

Bug Fix

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1500

How should this be tested?

Outline the steps to test the PR here.

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

felixcheung · 2016-09-27T23:21:15Z

why is pythonScriptInitialized check removed?

astroshim · 2016-09-28T00:38:18Z

@felixcheung Thank you for quick review.
It's because we should get result message even if pythonScriptInitialized is true and I thought pythonScriptInitialized on the completion() increases the complexity of code.

felixcheung · 2016-09-28T04:52:11Z

do we have tests for completion?

fanlu · 2016-09-28T15:24:47Z

I use git apply this patch
but the problem is still there
ERROR [2016-09-28 23:23:53,892]({qtp659748578-108} NotebookServer.java[onMessage]:261) - Can't handle message
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:453)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:122)
at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:242)
at org.apache.zeppelin.notebook.Note.completion(Note.java:520)
at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:797)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:217)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:362)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:342)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:449)
... 21 more

fanlu · 2016-09-28T15:29:16Z

@astroshim Another problem has emerged
ERROR [2016-09-28 23:29:20,018]({pool-1-thread-2} TThreadPoolServer.java[run]:296) - Error occurred during processing of message.
java.lang.NullPointerException
at org.apache.zeppelin.spark.PySparkInterpreter.completion(PySparkInterpreter.java:453)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:122)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.completion(RemoteInterpreterServer.java:469)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1742)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1727)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

astroshim · 2016-09-29T02:24:01Z

@fanlu That error seems like occurs before launching pyspark interpreter.
Could you please try completion after launching pyspark interpreter if works or not like following?

fanlu · 2016-09-29T03:31:44Z

@astroshim It worked in pyspark, thanks
How about python autocomplete
ERROR [2016-09-29 11:35:43,314]({qtp659748578-207} NotebookServer.java[onMessage]:261) - Can't handle message
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:453)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:122)
at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:242)
at org.apache.zeppelin.notebook.Note.completion(Note.java:520)
at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:797)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:217)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:362)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:342)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:449)
... 21 more

…into ZEPPELIN-1500

astroshim · 2016-11-08T14:04:17Z

rebuild CI

astroshim · 2016-11-09T02:26:42Z

re-trigger CI

astroshim · 2016-11-09T09:37:52Z

please review this PR.

felixcheung · 2016-11-09T19:36:21Z

LGTM, tests are green

Leemoonsoo · 2016-11-10T14:36:10Z

LGTM and there're no further discussions

fix bug of condition check.

ee2a29b

astroshim mentioned this pull request Sep 27, 2016

ZEPPELIN-497 ] pyspark completion #530

Closed

fix condition

60da1c3

remove NullPointException

49282b7

astroshim closed this Nov 8, 2016

astroshim added 4 commits November 8, 2016 18:24

Merge branch 'ZEPPELIN-1500' of https://github.com/astroshim/zeppelin …

7e93968

…into ZEPPELIN-1500

add testcase

cb4b0fb

Merge branch 'master' into ZEPPELIN-1500

e50606a

fix to wait completion data

d442c12

astroshim reopened this Nov 8, 2016

astroshim closed this Nov 8, 2016

astroshim reopened this Nov 8, 2016

astroshim closed this Nov 9, 2016

astroshim reopened this Nov 9, 2016

asfgit closed this in c4e722a Nov 11, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-1500] pyspark completion doesn't work.#1465

[ZEPPELIN-1500] pyspark completion doesn't work.#1465
astroshim wants to merge 7 commits intoapache:masterfrom
astroshim:ZEPPELIN-1500

astroshim commented Sep 27, 2016

Uh oh!

felixcheung commented Sep 27, 2016

Uh oh!

astroshim commented Sep 28, 2016

Uh oh!

felixcheung commented Sep 28, 2016

Uh oh!

fanlu commented Sep 28, 2016

Uh oh!

fanlu commented Sep 28, 2016

Uh oh!

astroshim commented Sep 29, 2016

Uh oh!

fanlu commented Sep 29, 2016 •

edited

Loading

Uh oh!

astroshim commented Nov 8, 2016

Uh oh!

astroshim commented Nov 9, 2016

Uh oh!

astroshim commented Nov 9, 2016

Uh oh!

felixcheung commented Nov 9, 2016 •

edited

Loading

Uh oh!

Leemoonsoo commented Nov 10, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

astroshim commented Sep 27, 2016

What is this PR for?

What type of PR is it?

What is the Jira issue?

How should this be tested?

Questions:

Uh oh!

felixcheung commented Sep 27, 2016

Uh oh!

astroshim commented Sep 28, 2016

Uh oh!

felixcheung commented Sep 28, 2016

Uh oh!

fanlu commented Sep 28, 2016

Uh oh!

fanlu commented Sep 28, 2016

Uh oh!

astroshim commented Sep 29, 2016

Uh oh!

fanlu commented Sep 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

astroshim commented Nov 8, 2016

Uh oh!

astroshim commented Nov 9, 2016

Uh oh!

astroshim commented Nov 9, 2016

Uh oh!

felixcheung commented Nov 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Leemoonsoo commented Nov 10, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fanlu commented Sep 29, 2016 •

edited

Loading

felixcheung commented Nov 9, 2016 •

edited

Loading